Structured graph-to-text generation with two step fine-tuning

ABSTRACT

Embodiments described herein provide systems and methods for data-to-text generation. The embodiments receive input data that includes a resource description framework (RDF) triples in an RDF graph. A data-to-text generation system generates position aware embeddings, including position embeddings, triple role embeddings, and tree-level embeddings. Using the position aware embeddings and the RDF graph, the data-to-text generation system generates a textual description for the RDF graph.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a nonprovisional of and claims priority under 35U.S.C. 119 to U.S. provisional Application No. 63/065,965, filed Aug.14, 2020, which is hereby expressly incorporated by reference herein inits entirety.

TECHNICAL FIELD

The disclosure relates generally to data-to-text generation systemsbased on machine learning, and more specifically to systems thatgenerate a description from structured input data using position awareembeddings.

BACKGROUND

Data-to-text generation is treated as a graph-to-text generation task,where a model receives a complex knowledge graph and generates afaithful description. Currently neural network-based machine learningmethods are divided into two categories: end-to-end data-to-textgeneration which directly generate descriptions from input knowledgegraphs, and a two-step generation methods which try to controlgeneration quality by first explicitly reasoning about the underlyingstructure. A conventional end-to-end data-to-text generation moduleutilizes large pretrained language model to generate text. However, theend-to-end data-to-text generation module may generate fabricated factsfrom the pretrained language model or incorrectly divide the triples.Moreover, when receiving the structured input, some of the models maysimply concatenate those triples together. Other models use complicatedgraph neural networks that encode the triple structures for generation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a computing device for implementing adata-to-text generation system, according to some embodiments.

FIG. 2 is an example structured input data, according to someembodiments.

FIG. 3 is an example textual description generated from the structuredinput data, according to some embodiments.

FIG. 4 is diagram illustrating position aware embeddings, according tosome embodiments.

FIG. 5 is a block diagram of a data-to-text module, according to someembodiments.

FIG. 6 is a flowchart of a method for generating a textual descriptionusing a data-to-text generation system, according to some embodiments.

FIG. 7 illustrates a table that illustrates results of the data-to-textgeneration system compared to conventional data-to-text generationsystems.

In one or more implementations, not all of the depicted components ineach figure may be required, and one or more implementations may includeadditional components not shown in a figure. Variations in thearrangement and type of the components may be made without departingfrom the scope of the subject disclosure. Additional components,different components, or fewer components may be utilized within thescope of the subject disclosure.

DETAILED DESCRIPTION

Various embodiments are directed to data-to-text generation systems thatmay generate a textual description from structured input data. Morespecifically, given a structured input data, such as a set of resourcedescription framework (RDF) triples or a Wikipedia infobox in the formof trees or graphs, the embodiments may generate corresponding textdescriptions. In accordance with the disclosure herein, the data-to-textgeneration system may be trained to generate position aware embeddingsfor the structured input data. The position aware embeddings may helpthe data-to-text generation system to fully capture input structuressuch as a word position and its role in the structured data, location ofthe triple in the structured data, and tree-level order of the triple insome embodiments.

The embodiments are also directed to pre-training a generative languagemodel with position aware embeddings. The position aware embeddings helpthe linearized knowledge graph to more flexibly encode the graphstructure and external knowledge such as entity type information fromthe background data dumps, such as Wikipedia dumps.

As used herein, the term “network” may comprise any hardware orsoftware-based framework that includes any artificial intelligencenetwork or system, neural network or system and/or any training orlearning models implemented thereon or therewith.

As used herein, the term “module” may comprise hardware orsoftware-based framework that performs one or more functions. In someembodiments, the module may be implemented on one or more neuralnetworks, such as supervised or unsupervised neural networks,convolutional neural networks, or memory-augmented neural networks,among others.

FIG. 1 is a simplified diagram of a computing device 100 forimplementing data-to-text generation system, according to someembodiments. Computing device 100 includes processor 110 and memory 120.Memory 120 includes a data-to-text generation system 130. In someexamples, data-to-text generation system 130 may be used to receive andhandle input 140. The input may be structured data, such as a resourcedescription framework (RDF) graph discussed below. Data-to-textgeneration system 130 may include a training module 160 and data-to-textmodule 170. Training module 160 and data-to-text module 170 may furtherinclude additional submodules, such as those described in to FIGS. 2-6or may be implemented in hardware, software, and/or a combination ofhardware and software. Both training module 160 and data-to-text module170 may be implemented as one or more neural network models andcomponents thereof.

As shown in FIG. 1 , processor 110 is coupled to memory 120. Operationof computing device 100 is controlled by processor 110. And althoughcomputing device 100 is shown with only one processor 110, it isunderstood that processor 110 may be representative of one or morecentral processing units (CPUs), multi-core processors, microprocessors,microcontrollers, and/or the like in computing device 100. Althoughprocessor 110 may include one or more general purpose central processingunits (CPUs), processor 110 may additionally or alternately include atleast one processor that provides accelerated performance whenevaluating neural network models. For example, processor 110 may includea graphics processing unit (GPU), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA), a tensorprocessing unit (TPU), a digital signal processor (DSP), asingle-instruction multiple-data (SIMD) processor, and/or the like.Generally, such processors may accelerate various computing tasksassociated with evaluating neural network models (e.g., training,prediction, preprocessing, and/or the like) by an order of magnitude ormore in comparison to a general-purpose CPU. Computing device 100 may beimplemented as a stand-alone subsystem, as a board added to a computingdevice, and/or as a virtual machine.

Processor 110 and/or memory 120 may be arranged in any suitable physicalarrangement. In some embodiments, processor 110 and/or memory 120 may beimplemented on a same board, in a same package (e.g.,system-in-package), on a same chip (e.g., system-on-chip), and/or thelike. In some embodiments, processor 110 and/or memory 120 may includedistributed, virtualized, and/or containerized computing resources.Consistent with such embodiments, processor 110 and/or memory 120 may belocated in one or more data centers and/or cloud computing facilities.

Memory 120 may be used to store instructions executable by computingdevice 100 and/or one or more data structures used during operation ofcomputing device 100. Memory 120 may include one or more types ofmachine-readable media. In some examples, memory 120 may includenon-transitory, tangible, machine-readable media that includesexecutable code that when run by one or more processors (e.g., processor110) may cause the one or more processors to perform the methodsdescribed in further detail herein. Memory 120 may include various typesof short-term and/or long-term storage modules including cache memory,random access memory (RAM), static random access memory (SRAM), dynamicrandom access memory (DRAM), non-volatile memory (NVM), flash memory,solid state drives (SSD), hard disk drive (HDD), optical storage media,magnetic tape, any other memory chip or cartridge, and/or any othermedium from which a processor or computer is adapted to read. Somecommon forms of machine-readable media may include flexible disk, harddisk, magnetic tape, any other magnetic medium, compact disk read-onlymemory (CD-ROM), any other optical medium, punch cards, paper tape, anyother physical medium with patterns of holes, programmable read-onlymemory (PROM), erasable programmable read-only memory (EPROM),electrically erasable programmable read-only memory (EEPROM), any othermemory chip or cartridge, and/or any other medium from which a processoror computer is adapted to read.

In some embodiments, memory 120 includes instructions for data-to-textgeneration system 130 that may be used to implement and/or emulate thesystems and models, and/or to implement any of the methods describedfurther herein. The data-to-text generation system 130 may correspond toa neural network model that is evaluated by processor 110. Inparticular, the data-to-text generation system 130 may include aplurality of neural network layers. Examples of neural network layersinclude densely connected layers, convolutional layers, recurrentlayers, pooling layers, dropout layers, and/or the like. In someembodiments, the data-to-text generation system 130 may include at leastone hidden layer that is not directly connected to either an input or anoutput of the neural network. The data-to-text generation system 130 mayfurther include a plurality of model parameters (e.g., weights and/orbiases) that are learned according to a machine learning process.Examples of machine learning processes include supervised learning,reinforcement learning, unsupervised learning, and/or the like.

Computing device 100 may receive input 140, which may be structureddata, such as an RDF graph. Input 140 may be provided to thedata-to-text generation system 130. The data-to-text generation system130 operates on the input 140 to generate an output 150. Output 150 maybe a textual description of the input 140, e.g. textual description ofthe RDF graph. Although the description below is discussed in terms ofRDF, the embodiments equally apply to other types of structured data.

FIG. 2 illustrates a diagram of an example resource descriptionframework (RDF) graph 200, according to some embodiments. The underlyingstructure of any expression in an RDF graph 200 may be a collection oftriples. Each tripe may include a subject (S), a predicate (P) and anobject (O). The RDF graph 200 may be stored as a data structure in amemory of a computer discussed in FIG. 1 . Each triple in the RDF graph200 may be in a {S, P, O} format, where the subject and object are nodesin the RDF graph 200 and the predicate is a link between the two nodes.Some example triples from the RDF graph 200 in FIG. 2 may be{“Karnataka”, “state”, “Acharya Institute of Technology”}, {“AcharyaInstitute of Technology”, “sports offered”, “location”}, {“Tennis”,“sports governing body”, “International Tennis Federation”}, and{“Telangana”, “northeast”, “Karnataka”}.

FIG. 3 illustrates a diagram 300 of a textual description that thedata-to-text generation system may generate from an RDF graph, accordingto some embodiments. The output illustrated in FIG. 3 may be output 150that is a textual description of RDF graph 200 illustrated in FIG. 2 .Notably, the textual description includes words in the triples of RDFgraph 200.

Going back to FIG. 1 , data-to-text module 170 in data-to-textgeneration system 130 may be or include a pre-trained language model.Some nonlimiting examples of the pre-trained language model may be BART,distill-BART-xsum, and T5 or their variants. The pre-trained languagemodels may include one or more embedding layers, encoders, decoders,attention layers and/or classifiers in some embodiments. As discussedabove, data-to-text module 170 may receive structured data and generatea textual description of the structured data.

In some embodiments, training module 160 may train data-to-text module170 using known structured datasets, such as WebNLG dataset or aWikipedia Corpus. Unlike conventional language models, training module160 may also train data-to-text module 170 to generate and processdifferent types of embeddings, including position aware embeddings.Example embeddings may be token embeddings. Example position awareembeddings may be position embeddings, triple role embeddings, andtree-level embeddings. FIG. 4 illustrates a block diagram 400 ofembeddings, according to some embodiments. These embeddings may includetoken embeddings, position embeddings, triple role embeddings, andtree-level embeddings.

In some embodiments, token embeddings may include embeddings for tokensthat correspond to words (entities and relations) in RDF graph 200 aswell as embeddings for tokens with an indication that the token stores asubject, a predicate, or an object. Typically, there may be one tokenembeddings per word, and one token for an indicator that indicateswhether the word or words are associated with a subject (S), predicate(P), or object (O). For example, suppose RDF graph 200 includes aportion of a triple where a subject (S) is “Ajoblanco Advances” andpredicate (P) is “editor.” FIG. 4 illustrates that for this case, thetoken embeddings may include token embeddings for words “Ajoblanco,”“Advances,” and “editor,” as well as token embeddings for the subject(S|) and predicate (P|). For example, token embedding for subject (S|)indicate that the following tokens stores subjects, such as “Ajoblanco”and “Advances.” Additionally, a special [CLS] token may be concatenatedto the beginning of the first triple, to indicate a beginning of the RDFgraph 200. Training module 160 may train data-to-text module 170 togenerate a token embedding for the special [CLS] token.

In some embodiments, position embeddings may include a positionidentifier (ID). The position ID is an index of the token in theflattened RDF graph 200 sequence. For example, the position embeddingfor the [CLS] token may indicate that token [CLS] is in a zerothposition, that is Position ID=0 (if the position count begins withzero), position embedding for token “S|” may indicate that the token isin the first position, that is position ID=1, and position embedding fortoken “Ajoblanco” may indicate that the token is in the second position,that is position ID=2.

In some embodiments, triple role embeddings may include a triple role IDthat differentiates different triple roles. The triple role ID may beset to one for a subject indicator “S|” and words that correspond to thesubject, to two for a relation indicator (e.g. predicate “P|”) and wordsthat correspond to the relation, and three for an object indicator(“O|”) and words that correspond to an object, in some embodiments. Withreference to FIG. 4 , triple role ID may be set to one for tokens “S|,”“Ajoblanco,” and “Advances” for the subject in the triple, and triplerole ID may be set to two for tokens “P|” and “editor” for a relation inthe triple.

In some embodiments, the tree level embeddings may include a tree levelID. The tree level ID may indicate the distance, e.g. number ofrelations from the root of the parsing tree that may store parsed RDFgraph. With reference to FIG. 4 , tree level ID for all tokens in thetriple may be set to two and tree level ID may be set to zero for thespecial [CLS] token.

Going back to FIG. 1 , in some embodiments, training module 160 maypretrain the data-to-text module 170 using a dataset, such as aWikipedia dataset that includes wikidata-description pairs. Thewikidata-description pairs may be RDF graph-description pairs. Topretrain the data-to-text module 170, training module 160 may createdifferent categories in the dataset and select the graph-descriptionpairs for a particular category. Training module 160 may then linearizethe dataset in the particular category by prepending tokens such as[CLS], S|, P|, and O| to the dataset and pass the dataset throughdata-to-text module 170 that is trained to recognize the tokenembeddings, position embeddings, triple role embeddings, and tree-levelembeddings. Once data-to-text module 170 is trained, data-to-text module170 may be used to generate text description for an RDF graph.

FIG. 5 illustrates a block diagram 500 of an example model architecturefor a data-to-text module, according to some embodiments. Data-to-textmodule 170 may receive RDF graph 200. Once received, data-to-text module170 may generate embeddings, such as token embeddings 505, and positionaware embeddings, such as position embeddings 510, triple roleembeddings 515, and tree-level embeddings 520 for the RDF graph 200.Using token embeddings 505, position embeddings 510, triple roleembeddings 515, and tree-level embeddings 520, data-to-text module 405may use its pre-trained generative language model to determine a textualdescription 530 for RDF graph 200, such as the output illustrated inFIG. 3 .

FIG. 6 is a simplified diagram of a method 600 for generating adescription using a data-to-text generation system, according to someembodiments. One or more of the processes 602-606 of method 600 may beimplemented, at least in part, in the form of executable code stored onnon-transitory, tangible, machine-readable media that when run by one ormore processors may cause the one or more processors to perform one ormore of the processes 602-606.

At process 602, a structured data, such as an RDF graph that includestriples is received. As discussed above, data-to-text generation system130 may receive structured data, such as RDF graph 200.

At process 604, embeddings are generated. For example, data-to-textgeneration system 130 generates token embeddings 505, and position awareembeddings, such as position embeddings 510, triple role embeddings 515,and tree-level embeddings 520 from triples in RDF graph 200.

At process 608, a description is generated. For example, the pre-trainedgenerative language model of data-to-text module 170 may generate atextual description, such as output 150 shown in FIG. 3 of structureddata, e.g., RDF graph 200, using the triples in the RDF graph 200 andthe embeddings generated in process 604.

FIG. 7 illustrates a table 700 that includes results of the data-to-textgeneration system compared to conventional data-to-text generationsystems. The pre-trained generative language models included indata-to-text module 170 may be the BART-base model, the BART-xsum-12-6model or the T5-base model trained using training module 160 to includetoken embeddings and position aware embeddings. Additionally, theBART-base model, the BART-xsum-12-6 model or the T5-base model may betrained using a structured dataset, such as the Wikipedia dataset. Theresults of the data-to-text generation system 130 that includesdata-to-text module 170 with the pre-trained generative language modelsdiscussed above is compared against conventional generative languagemodels. These conventional models are Pipeline GRU, PipelineTransformer, MELBOURNE, GCN-EC, BestPlan, PlanEnc, R5-base, and T5-largemodel. The BLUE, METEOR, and TER scripts were used to generate theresults. As illustrated in FIG. 7 , the data-to-text module 170 thatincludes the BART-base, Bart-large, distill-BART-xsum, T5-base, andT5-large that are trained using training module 160 generated superiorresults and outperform conventional generative language models.

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or applications should not betaken as limiting. Various mechanical, compositional, structural,electrical, and operational changes may be made without departing fromthe spirit and scope of this description and the claims. In someinstances, well-known circuits, structures, or techniques have not beenshown or described in detail in order not to obscure the embodiments ofthis disclosure Like numbers in two or more figures represent the sameor similar elements.

In this description, specific details are set forth describing someembodiments consistent with the present disclosure. Numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments. It will be apparent, however, to one skilled in the artthat some embodiments may be practiced without some or all of thesespecific details. The specific embodiments disclosed herein are meant tobe illustrative but not limiting. One skilled in the art may realizeother elements that, although not specifically described here, arewithin the scope and the spirit of this disclosure. In addition, toavoid unnecessary repetition, one or more features shown and describedin association with one embodiment may be incorporated into otherembodiments unless specifically described otherwise or if the one ormore features would make an embodiment non-functional.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

What is claimed is:
 1. A system comprising: a non-transitory memory; andone or more hardware processors coupled to the non-transitory memory andconfigured to read instructions from the non-transitory memory to causethe system to perform operations comprising: receiving, at adata-to-text generation system that includes a generative languagemodel, an input data that includes resource description framework (RDF)triples in an RDF graph; generating, using the data-to-text generationsystem, embeddings from the RDF graph based on tokens of the input data,wherein the embeddings include a position aware embedding thatidentifies a position of an RDF triple of the RDF triples in the RDFgraph; and generating, using the data-to-text generation system, atextual description of the input data based on the embeddings and theRDF graph, wherein the position aware embedding includes a positionembedding that identifies a position of a token indicating whether aword in the RDF triple of the RDF triples is a subject, a relation, oran object.
 2. The system of claim 1, wherein the RDF triple includeswords that correspond to a subject, a relation, or an object.
 3. Thesystem of claim 1, wherein the position aware embedding includes aposition embedding that identifies a position of a token that stores aword in the RDF triple from the RDF triples.
 4. The system of claim 1,wherein the position aware embedding includes a triple role embeddingthat identifies that a token includes a word or an indication of a roleof the word in the RDF triple from the RDF triples that corresponds to asubject, an object, or a relation.
 5. The system of claim 1, wherein theposition aware embedding includes a tree-level embedding that identifiesa tree distance from a root of a parsing tree to a level in the parsingtree that includes a token, wherein the token stores a word or anindication of a role of the word in the RDF triple from the RDF triples.6. The system of claim 1, wherein the generating the embeddings furthercomprises generating a token embedding that identifies a token thatstores a word or an indication of a role of the word in the RDF triplefrom the RDF triples.
 7. A method comprising: receiving, at adata-to-text generation system that includes a generative languagemodel, the data-to-text generation system configured to execute on aprocessor, an input data that includes resource description framework(RDF) triples in an RDF graph; generating, using the data-to-textgeneration system, embeddings from the RDF graph based on tokens of theinput data, wherein the embeddings include a position aware embeddingthat identifies a position of an RDF triple of the RDF triples in theRDF graph; and generating, using the data-to-text generation system, atextual description of the input data based on the position awareembedding and the RDF graph, wherein the position aware embeddingincludes a position embedding that identifies a position of a tokenindicating whether a word in the RDF triple of the RDF triples is asubject, a relation, or an object.
 8. The method of claim 7, furthercomprising: training the generative language model to generate theposition aware embeddings.
 9. The method of claim 7, wherein theposition aware embedding includes a position embedding that identifies aposition of a token that stores a word in the RDF triple from the RDFtriples.
 10. The method of claim 7, wherein the position aware embeddingincludes a triple role embedding that identifies that a token includes aword or an indication of a role of the word in the RDF triple from theRDF triples that corresponds to a subject, an object, or a relation. 11.The method of claim 7, wherein the position aware embedding includes atree-level embedding that identifies a tree distance from a root of aparsing tree to a level in the parsing tree that includes a token,wherein the token stores a word or an indication of a role of the wordin the RDF triple from the RDF triples.
 12. The method of claim 7,wherein the generating the embeddings further comprises generating atoken embedding that identifies a token that stores a word or anindication of a role of the word in the RDF triple from the RDF triples.13. A non-transitory machine-readable medium having stored thereonmachine-readable instructions executable to cause a machine to performoperations comprising: receiving, at a data-to-text generator thatincludes a generative language model, an input data that includesstructured data triples in a structured graph; generating, using thedata-to-graph generator, embeddings from the structured graph based ontokens of the input data, wherein the embeddings include position awareembeddings that identify position of a triple of triples in thestructured graph; and generating, using a data-to-text module, a textualdescription of the input data based on the position aware embeddings andthe structured graph, wherein the position aware embeddings include aposition embedding that identifies a position of a token indicatingwhether a word in the triple of the triples is a subject, a relation, oran object.
 14. The non-transitory machine-readable medium of claim 13,wherein the position aware embeddings include a position embedding thatidentifies a position of a token that stores a word in the triple fromthe triples.
 15. The non-transitory machine-readable medium of claim 13,wherein the position aware embeddings include a triple role embeddingthat identifies that a token includes a word or an indication of a roleof the word in the triple from the triples that corresponds to asubject, an object, or a relation.
 16. The non-transitorymachine-readable medium of claim 13, wherein the position awareembeddings include a tree-level embedding that identifies a treedistance from a root of a parsing tree to a level in the parsing treethat stores a token, wherein the token includes a word or an indicationof a role of the word in the triple from the triples.
 17. Thenon-transitory machine-readable medium of claim 13, wherein thegenerating the embeddings further comprises generating token embeddingsthat identify a token that stores a word or an indication of a role ofthe word in the triple from the triples.